noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
This repository provides optimized serving configurations for the Qwen3.8-27B model running on a single 24 GB consumer GPU (RTX 3090). It achieves high throughput of approximately 1,000 tok/s at 64 concurrent users in batch mode and up to ~133 tok/s for single-user scenarios using speculative decoding techniques like MTP or DFlash2. The project includes custom patches, requantization scripts (such as int8 tensor-core GEMMs), and Docker support to enable extended context windows of 150k to 262k tokens on a single consumer card.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
Arsen Apostolov writes about the actual electrical cost of running local Large Language Models on a single NVIDIA RTX 3090 compared to hosted cloud APIs.
>"I measured the actual GPU electricity for eight local models on one RTX 3090 — and the cheapest wasn't the smallest, nor the priciest the biggest"
Cost of Generating 1 Million Tokens Locally
| MODEL | PARAMS (Billions) | MEAN SPEED (tok/s) | AVG GPU DRAW (W) | € / 1M OUTPUT TOKENS |
| :--- | :---: | :---: | :---: | :---: |
| **gemma3:1b** | 1B | 136 tok/s | 154 W | €0.060 |
| **Qwen3-Coder** | 30.5B | 130 tok/s | 233 W | €0.112 |
| **gemma4:26b** | 26B | 85 tok/s | 246 W | €0.139 |
| **Devstral** | 24B | 49 tok/s | 320 W | €0.321 |
| **gemma3:27b** | 27B | 36 tok/s | 283 W | €0.361 |
| **Seed-OSS** | 36B | 4.5 tok/s | 186 W | €0.946 |
| **GLM-4.5-Air** | 106B | 5.7 tok/s | 141 W | €1.040 |
| **DeepSeek-R1-Distill** | 32.8B | 6.9 tok/s | 155 W | €1.526 |
By measuring real-time GPU power consumption through a custom dashboard, he discovered that token costs are driven by effective wall-clock throughput rather than model parameter size or raw generation speed alone. The results show that while small and fast models can be more economical than cloud services, reasoning-heavy models may actually become the most expensive to run locally due to the time spent "deliberating" between tokens.
* Measurements were performed using HomeLab Monitor, an open-source dashboard that integrates live power data from `nvidia-smi`.
* DeepSeek-R1-Distill emerged as the most expensive model per million tokens because its effective throughput is slowed by reasoning delays.
* The findings focus on marginal electricity costs and exclude total cost of ownership factors like hardware amortization or idle draw.
>"I measured the actual GPU electricity for eight local models on one RTX 3090 — and the cheapest wasn't the smallest, nor the priciest the biggest"
Cost of Generating 1 Million Tokens Locally
| MODEL | PARAMS (Billions) | MEAN SPEED (tok/s) | AVG GPU DRAW (W) | € / 1M OUTPUT TOKENS |
| :--- | :---: | :---: | :---: | :---: |
| **gemma3:1b** | 1B | 136 tok/s | 154 W | €0.060 |
| **Qwen3-Coder** | 30.5B | 130 tok/s | 233 W | €0.112 |
| **gemma4:26b** | 26B | 85 tok/s | 246 W | €0.139 |
| **Devstral** | 24B | 49 tok/s | 320 W | €0.321 |
| **gemma3:27b** | 27B | 36 tok/s | 283 W | €0.361 |
| **Seed-OSS** | 36B | 4.5 tok/s | 186 W | €0.946 |
| **GLM-4.5-Air** | 106B | 5.7 tok/s | 141 W | €1.040 |
| **DeepSeek-R1-Distill** | 32.8B | 6.9 tok/s | 155 W | €1.526 |
By measuring real-time GPU power consumption through a custom dashboard, he discovered that token costs are driven by effective wall-clock throughput rather than model parameter size or raw generation speed alone. The results show that while small and fast models can be more economical than cloud services, reasoning-heavy models may actually become the most expensive to run locally due to the time spent "deliberating" between tokens.
* Measurements were performed using HomeLab Monitor, an open-source dashboard that integrates live power data from `nvidia-smi`.
* DeepSeek-R1-Distill emerged as the most expensive model per million tokens because its effective throughput is slowed by reasoning delays.
* The findings focus on marginal electricity costs and exclude total cost of ownership factors like hardware amortization or idle draw.
The RTX 3090 offers a compelling combination of performance and 24GB of VRAM, making it a better choice for local LLM and AI workloads than newer Nvidia Blackwell GPUs like the RTX 5070 and even the RTX 5080, due to VRAM limitations and pricing.
A user shares their optimal settings for running the gpt-oss-120b model on a system with dual RTX 3090 GPUs and 128GB of RAM, aiming for a balance between performance and quality.
A user shares their experience running the GPT-OSS 120b model on Ollama with an i7 6700, 64GB DDR4 RAM, RTX 3090, and a 1TB SSD. They note slow initial token generation but acceptable performance overall, highlighting it's possible on a relatively modest setup. The discussion includes comparisons to other hardware configurations, optimization techniques (llama.cpp), and the model's quality.
>I have a 3090 with 64gb ddr4 3200 RAM and am getting around 50 t/s prompt processing speed and 15 t/s generation speed using the following:
>
>`llama-server -m <path to gpt-oss-120b> --ctx-size 32768 --temp 1.0 --top-p 1.0 --jinja -ub 2048 -b 2048 -ngl 99 -fa 'on' --n-cpu-moe 24`
> This about fills up my VRAM and RAM almost entirely. For more wiggle room for other applications use `--n-cpu-moe 26`.
>I have a 3090 with 64gb ddr4 3200 RAM and am getting around 50 t/s prompt processing speed and 15 t/s generation speed using the following:
>
>`llama-server -m <path to gpt-oss-120b> --ctx-size 32768 --temp 1.0 --top-p 1.0 --jinja -ub 2048 -b 2048 -ngl 99 -fa 'on' --n-cpu-moe 24`
> This about fills up my VRAM and RAM almost entirely. For more wiggle room for other applications use `--n-cpu-moe 26`.
Learn how to use a spare GPU to create an external graphics card (eGPU) for your laptop or PC gaming handheld, including using prebuilt enclosures, DIY Thunderbolt enclosures, or OCuLink enclosures.
Backprop provides powerful and affordable GPU instances for AI development, with pre-built environments, pay-as-you-go pricing, and fast internet.
A startup called Backprop has demonstrated that a single Nvidia RTX 3090 GPU, released in 2020, can handle serving a modest large language model (LLM) like Llama 3.1 8B to over 100 concurrent users with acceptable throughput. This suggests that expensive enterprise GPUs may not be necessary for scaling LLMs to a few thousand users.
How to get oobabooga/text-generation-webui running on Windows or Linux with LLaMa-30b 4bit mode via GPTQ-for-LLaMa on an RTX 3090 start to finish.
